{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "e67dbc8d-2509-4253-9c64-cd25d1c6f380"
   },
   "source": [
    "# 推荐系统组队学习 第四次打卡：特征工程\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "b8c1f75d-0bcd-4437-baa8-80a49be3c803"
   },
   "source": [
    "### 可以直接使用的特征：\n",
    "文章的自身特征， category_id表示这文章的类型， created_at_ts表示文章建立的时间， 这个关系着文章的时效性，\n",
    "\n",
    "words_count是文章的字数， 一般字数太长我们不太喜欢点击, 也不排除有人就喜欢读长文。\n",
    "\n",
    "文章的内容embedding特征， 这个召回的时候用过， 这里可以选择使用， 也可以选择不用， 也可以尝试其他类型的embedding特征， 比如W2V等\n",
    "\n",
    "用户的设备特征信息\n",
    "\n",
    "构造监督数据集的思路， 根据召回结果， 我们会得到一个{user_id: [可能点击的文章列表]}形式的字典。 那么我们就可以对于每个用户， 每篇可能点击的文章构造一个监督测试集， 比如对于用户user1， 假设得到的他的召回列表{user1: [item1, item2, item3]}， 我们就可以得到三行数据(user1, item1), (user1, item2), (user1, item3)的形式， 这就是监督测试集时候的前两列特征。\n",
    "\n",
    "构造特征的思路是这样， 我们知道每个用户的点击文章是与其历史点击的文章信息是有很大关联的， 比如同一个主题， 相似等等。 所以特征构造这块很重要的一系列特征是要结合用户的历史点击文章信息。我们已经得到了每个用户及点击候选文章的两列的一个数据集， 而我们的目的是要预测最后一次点击的文章， 比较自然的一个思路就是和其最后几次点击的文章产生关系， 这样既考虑了其历史点击文章信息， 又得离最后一次点击较近，因为新闻很大的一个特点就是注重时效性。 往往用户的最后一次点击会和其最后几次点击有很大的关联。 所以我们就可以对于每个候选文章， 做出与最后几次点击相关的特征如下：\n",
    "\n",
    "候选item与最后几次点击的相似性特征(embedding内积） — 这个直接关联用户历史行为\n",
    "候选item与最后几次点击的相似性特征的统计特征 — 统计特征可以减少一些波动和异常\n",
    "候选item与最后几次点击文章的字数差的特征 — 可以通过字数看用户偏好\n",
    "候选item与最后几次点击的文章建立的时间差特征 — 时间差特征可以看出该用户对于文章的实时性的偏好\n",
    "还需要考虑一下  \n",
    "如果使用了youtube召回的话， 我们还可以制作用户与候选item的相似特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "927e71e7-e822-4be9-9ee8-1bf06191ed6b"
   },
   "source": [
    "我们首先获得用户的最后一次点击操作和用户的历史点击， 这个基于我们的日志数据集做\n",
    "基于用户的历史行为制作特征， 这个会用到用户的历史点击表， 最后的召回列表， 文章的信息表和embedding向量\n",
    "制作标签， 形成最后的监督学习数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "uuid": "062d9341-2d37-4848-b417-56537f6c4f06"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.\n",
      "Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.\n",
      "To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.\n",
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Looking in indexes: http://yum.tbsite.net/pypi/simple/\n",
      "Collecting gensim\n",
      "  Downloading http://yum.tbsite.net/pypi/packages/2b/e0/fa6326251692056dc880a64eb22117e03269906ba55a6864864d24ec8b4e/gensim-3.8.3-cp36-cp36m-manylinux1_x86_64.whl (24.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 24.2 MB 9.7 MB/s \n",
      "\u001b[?25hRequirement already satisfied: six>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from gensim) (1.11.0)\n",
      "Collecting smart-open>=1.8.1\n",
      "  Downloading http://yum.tbsite.net/pypi/packages/e3/cf/6311dfb0aff3e295d63930dea72e3029800242cdfe0790478e33eccee2ab/smart_open-4.0.1.tar.gz (117 kB)\n",
      "\u001b[K     |████████████████████████████████| 117 kB 233 kB/s \n",
      "\u001b[?25hRequirement already satisfied: scipy>=0.18.1 in /opt/conda/lib/python3.6/site-packages (from gensim) (1.3.3)\n",
      "Requirement already satisfied: numpy>=1.11.3 in /opt/conda/lib/python3.6/site-packages (from gensim) (1.16.0)\n",
      "Building wheels for collected packages: smart-open\n",
      "  Building wheel for smart-open (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for smart-open: filename=smart_open-4.0.1-py3-none-any.whl size=115167 sha256=32e91871c4fad5ae6bd08e5336f8d342d34f24cff141cf408f6a417362a62f73\n",
      "  Stored in directory: /home/admin/.cache/pip/wheels/e2/e9/2b/d0351c31ec6063220777dd57375de68d644adc4d0fa74b439b\n",
      "Successfully built smart-open\n",
      "Installing collected packages: smart-open, gensim\n",
      "Successfully installed gensim-3.8.3 smart-open-4.0.1\n",
      "\u001b[33mWARNING: You are using pip version 20.2.4; however, version 20.3 is available.\n",
      "You should consider upgrading via the '/opt/conda/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "from tqdm import tqdm\n",
    "import gc, os\n",
    "import logging\n",
    "import time\n",
    "import lightgbm as lgb\n",
    "#!pip install gensim\n",
    "from gensim.models import Word2Vec\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "f0a14c29-fb0e-4fe2-87a4-9bdae12568ad"
   },
   "source": [
    "#### 使用节省内存的函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "uuid": "8779501d-4148-4ce2-87ea-c82690daabeb"
   },
   "outputs": [],
   "source": [
    "# 节省内存的一个函数\n",
    "# 减少内存\n",
    "def reduce_mem(df):\n",
    "    starttime = time.time()\n",
    "    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n",
    "    start_mem = df.memory_usage().sum() / 1024**2\n",
    "    for col in df.columns:\n",
    "        col_type = df[col].dtypes\n",
    "        if col_type in numerics:\n",
    "            c_min = df[col].min()\n",
    "            c_max = df[col].max()\n",
    "            if pd.isnull(c_min) or pd.isnull(c_max):\n",
    "                continue\n",
    "            if str(col_type)[:3] == 'int':\n",
    "                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n",
    "                    df[col] = df[col].astype(np.int8)\n",
    "                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n",
    "                    df[col] = df[col].astype(np.int16)\n",
    "                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n",
    "                    df[col] = df[col].astype(np.int32)\n",
    "                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n",
    "                    df[col] = df[col].astype(np.int64)\n",
    "            else:\n",
    "                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n",
    "                    df[col] = df[col].astype(np.float16)\n",
    "                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n",
    "                    df[col] = df[col].astype(np.float32)\n",
    "                else:\n",
    "                    df[col] = df[col].astype(np.float64)\n",
    "    end_mem = df.memory_usage().sum() / 1024**2\n",
    "    print('-- Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction),time spend:{:2.2f} min'.format(end_mem,\n",
    "                                                                                                           100*(start_mem-end_mem)/start_mem,\n",
    "                                                                                                           (time.time()-starttime)/60))\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "uuid": "cf9cf6f4-1476-4851-9096-b57a0bce2225"
   },
   "outputs": [],
   "source": [
    "data_path = './data_raw/'\n",
    "save_path = './temp_results/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "18b7c47c-1602-4f3e-9842-3be740177c11"
   },
   "source": [
    "#### 读取数据\n",
    "\n",
    "划分训练集与验证集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "uuid": "97050b1a-2e65-4869-b3db-0037799a0fb7"
   },
   "outputs": [],
   "source": [
    "# all_click_df指的是训练集\n",
    "# sample_user_nums 采样作为验证集的用户数量\n",
    "def trn_val_split(all_click_df, sample_user_nums):\n",
    "    all_click = all_click_df\n",
    "    all_user_ids = all_click.user_id.unique()\n",
    "    \n",
    "    # replace=True表示可以重复抽样，反之不可以\n",
    "    sample_user_ids = np.random.choice(all_user_ids, size=sample_user_nums, replace=False) \n",
    "    \n",
    "    click_val = all_click[all_click['user_id'].isin(sample_user_ids)]\n",
    "    click_trn = all_click[~all_click['user_id'].isin(sample_user_ids)]\n",
    "    \n",
    "    # 将验证集中的最后一次点击给抽取出来作为答案\n",
    "    click_val = click_val.sort_values(['user_id', 'click_timestamp'])\n",
    "    val_ans = click_val.groupby('user_id').tail(1)\n",
    "    \n",
    "    click_val = click_val.groupby('user_id').apply(lambda x: x[:-1]).reset_index(drop=True)\n",
    "    \n",
    "    # 去除val_ans中某些用户只有一个点击数据的情况，如果该用户只有一个点击数据，又被分到ans中，\n",
    "    # 那么训练集中就没有这个用户的点击数据，出现用户冷启动问题，给自己模型验证带来麻烦\n",
    "    val_ans = val_ans[val_ans.user_id.isin(click_val.user_id.unique())] # 保证答案中出现的用户再验证集中还有\n",
    "    click_val = click_val[click_val.user_id.isin(val_ans.user_id.unique())]\n",
    "    \n",
    "    return click_trn, click_val, val_ans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "c56e6400-6c17-451d-93f8-7a56cabe9a71"
   },
   "source": [
    "#### 获取历史点击和最后一次点击"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "uuid": "f7654c79-ab3d-4293-8c1c-8cc633b4f568"
   },
   "outputs": [],
   "source": [
    "# 获取当前数据的历史点击和最后一次点击\n",
    "def get_hist_and_last_click(all_click):\n",
    "    all_click = all_click.sort_values(by=['user_id', 'click_timestamp'])\n",
    "    click_last_df = all_click.groupby('user_id').tail(1)\n",
    "\n",
    "    # 如果用户只有一个点击，hist为空了，会导致训练的时候这个用户不可见，此时默认泄露一下\n",
    "    def hist_func(user_df):\n",
    "        if len(user_df) == 1:\n",
    "            return user_df\n",
    "        else:\n",
    "            return user_df[:-1]\n",
    "\n",
    "    click_hist_df = all_click.groupby('user_id').apply(hist_func).reset_index(drop=True)\n",
    "\n",
    "    return click_hist_df, click_last_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "9ad4ddb7-38a6-4ef0-a650-1c736982186b"
   },
   "source": [
    "#### 读取训练、验证及测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "uuid": "c58f7bfe-f290-426e-9ac5-0a534c9c0313"
   },
   "outputs": [],
   "source": [
    "def get_trn_val_tst_data(data_path, offline=True):\n",
    "    if offline:\n",
    "        click_trn_data = pd.read_csv(data_path+'train_click_log.csv')  # 训练集用户点击日志\n",
    "        click_trn_data = reduce_mem(click_trn_data)\n",
    "        click_trn, click_val, val_ans = trn_val_split(click_trn_data , sample_user_nums)\n",
    "    else:\n",
    "        click_trn = pd.read_csv(data_path+'train_click_log.csv')\n",
    "        click_trn = reduce_mem(click_trn)\n",
    "        click_val = None\n",
    "        val_ans = None\n",
    "    \n",
    "    click_tst = pd.read_csv(data_path+'testA_click_log.csv')\n",
    "    \n",
    "    return click_trn, click_val, click_tst, val_ans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "ec1215b5-98cc-4968-8ff8-818c15ea08e6"
   },
   "source": [
    "#### 读取召回列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "uuid": "a56f2036-d710-4686-9d9c-714b7253d5bf"
   },
   "outputs": [],
   "source": [
    "# 返回多路召回列表或者单路召回\n",
    "def get_recall_list(save_path, single_recall_model=None, multi_recall=False):\n",
    "    if multi_recall:\n",
    "        return pickle.load(open(save_path + 'final_recall_items_dict.pkl', 'rb'))\n",
    "    \n",
    "    if single_recall_model == 'i2i_itemcf':\n",
    "        return pickle.load(open(save_path + 'itemcf_recall_dict.pkl', 'rb'))\n",
    "    elif single_recall_model == 'i2i_emb_itemcf':\n",
    "        return pickle.load(open(save_path + 'itemcf_emb_dict.pkl', 'rb'))\n",
    "    elif single_recall_model == 'user_cf':\n",
    "        return pickle.load(open(save_path + 'youtubednn_usercf_dict.pkl', 'rb'))\n",
    "    elif single_recall_model == 'youtubednn':\n",
    "        return pickle.load(open(save_path + 'youtube_u2i_dict.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "uuid": "d52bf3f4-c50b-436d-8392-39a724ec68c0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m\n",
       "\u001b[0mWord2Vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m[\u001b[0m\u001b[0;34m'sentences=None'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'corpus_file=None'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'size=100'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'alpha=0.025'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'window=5'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'min_count=5'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'max_vocab_size=None'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'sample=0.001'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'seed=1'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'workers=3'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'min_alpha=0.0001'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'sg=0'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'hs=0'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'negative=5'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'ns_exponent=0.75'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'cbow_mean=1'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'hashfxn=<built-in function hash>'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'iter=5'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'null_word=0'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'trim_rule=None'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'sorted_vocab=1'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'batch_words=10000'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'compute_loss=False'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'callbacks=()'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'max_final_vocab=None'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mSource:\u001b[0m        \n",
       "\u001b[0;32mclass\u001b[0m \u001b[0mWord2Vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mBaseWordEmbeddingsModel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m\"\"\"Train, use and evaluate neural networks described in https://code.google.com/p/word2vec/.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    Once you're finished training a model (=no more updates, only querying)\u001b[0m\n",
       "\u001b[0;34m    store and use only the :class:`~gensim.models.keyedvectors.KeyedVectors` instance in `self.wv` to reduce memory.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    The model can be stored/loaded via its :meth:`~gensim.models.word2vec.Word2Vec.save` and\u001b[0m\n",
       "\u001b[0;34m    :meth:`~gensim.models.word2vec.Word2Vec.load` methods.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    The trained word vectors can also be stored/loaded from a format compatible with the\u001b[0m\n",
       "\u001b[0;34m    original word2vec implementation via `self.wv.save_word2vec_format`\u001b[0m\n",
       "\u001b[0;34m    and :meth:`gensim.models.keyedvectors.KeyedVectors.load_word2vec_format`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    Some important attributes are the following:\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    Attributes\u001b[0m\n",
       "\u001b[0;34m    ----------\u001b[0m\n",
       "\u001b[0;34m    wv : :class:`~gensim.models.keyedvectors.Word2VecKeyedVectors`\u001b[0m\n",
       "\u001b[0;34m        This object essentially contains the mapping between words and embeddings. After training, it can be used\u001b[0m\n",
       "\u001b[0;34m        directly to query those embeddings in various ways. See the module level docstring for examples.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    vocabulary : :class:`~gensim.models.word2vec.Word2VecVocab`\u001b[0m\n",
       "\u001b[0;34m        This object represents the vocabulary (sometimes called Dictionary in gensim) of the model.\u001b[0m\n",
       "\u001b[0;34m        Besides keeping track of all unique words, this object provides extra functionality, such as\u001b[0m\n",
       "\u001b[0;34m        constructing a huffman tree (frequent words are closer to the root), or discarding extremely rare words.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    trainables : :class:`~gensim.models.word2vec.Word2VecTrainables`\u001b[0m\n",
       "\u001b[0;34m        This object represents the inner shallow neural network used to train the embeddings. The semantics of the\u001b[0m\n",
       "\u001b[0;34m        network differ slightly in the two available training modes (CBOW or SG) but you can think of it as a NN with\u001b[0m\n",
       "\u001b[0;34m        a single projection and hidden layer which we train on the corpus. The weights are then used as our embeddings\u001b[0m\n",
       "\u001b[0;34m        (which means that the size of the hidden layer is equal to the number of features `self.size`).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.025\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwindow\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                 \u001b[0mmax_vocab_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1e-3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mseed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mworkers\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.0001\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                 \u001b[0msg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnegative\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mns_exponent\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.75\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcbow_mean\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhashfxn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhash\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnull_word\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                 \u001b[0mtrim_rule\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msorted_vocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mMAX_WORDS_IN_BATCH\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcompute_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                 \u001b[0mmax_final_vocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        sentences : iterable of iterables, optional\u001b[0m\n",
       "\u001b[0;34m            The `sentences` iterable can be simply a list of lists of tokens, but for larger corpora,\u001b[0m\n",
       "\u001b[0;34m            consider an iterable that streams the sentences directly from disk/network.\u001b[0m\n",
       "\u001b[0;34m            See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus`\u001b[0m\n",
       "\u001b[0;34m            or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples.\u001b[0m\n",
       "\u001b[0;34m            See also the `tutorial on data streaming in Python\u001b[0m\n",
       "\u001b[0;34m            <https://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/>`_.\u001b[0m\n",
       "\u001b[0;34m            If you don't supply `sentences`, the model is left uninitialized -- use if you plan to initialize it\u001b[0m\n",
       "\u001b[0;34m            in some other way.\u001b[0m\n",
       "\u001b[0;34m        corpus_file : str, optional\u001b[0m\n",
       "\u001b[0;34m            Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format.\u001b[0m\n",
       "\u001b[0;34m            You may use this argument instead of `sentences` to get performance boost. Only one of `sentences` or\u001b[0m\n",
       "\u001b[0;34m            `corpus_file` arguments need to be passed (or none of them, in that case, the model is left uninitialized).\u001b[0m\n",
       "\u001b[0;34m        size : int, optional\u001b[0m\n",
       "\u001b[0;34m            Dimensionality of the word vectors.\u001b[0m\n",
       "\u001b[0;34m        window : int, optional\u001b[0m\n",
       "\u001b[0;34m            Maximum distance between the current and predicted word within a sentence.\u001b[0m\n",
       "\u001b[0;34m        min_count : int, optional\u001b[0m\n",
       "\u001b[0;34m            Ignores all words with total frequency lower than this.\u001b[0m\n",
       "\u001b[0;34m        workers : int, optional\u001b[0m\n",
       "\u001b[0;34m            Use these many worker threads to train the model (=faster training with multicore machines).\u001b[0m\n",
       "\u001b[0;34m        sg : {0, 1}, optional\u001b[0m\n",
       "\u001b[0;34m            Training algorithm: 1 for skip-gram; otherwise CBOW.\u001b[0m\n",
       "\u001b[0;34m        hs : {0, 1}, optional\u001b[0m\n",
       "\u001b[0;34m            If 1, hierarchical softmax will be used for model training.\u001b[0m\n",
       "\u001b[0;34m            If 0, and `negative` is non-zero, negative sampling will be used.\u001b[0m\n",
       "\u001b[0;34m        negative : int, optional\u001b[0m\n",
       "\u001b[0;34m            If > 0, negative sampling will be used, the int for negative specifies how many \"noise words\"\u001b[0m\n",
       "\u001b[0;34m            should be drawn (usually between 5-20).\u001b[0m\n",
       "\u001b[0;34m            If set to 0, no negative sampling is used.\u001b[0m\n",
       "\u001b[0;34m        ns_exponent : float, optional\u001b[0m\n",
       "\u001b[0;34m            The exponent used to shape the negative sampling distribution. A value of 1.0 samples exactly in proportion\u001b[0m\n",
       "\u001b[0;34m            to the frequencies, 0.0 samples all words equally, while a negative value samples low-frequency words more\u001b[0m\n",
       "\u001b[0;34m            than high-frequency words. The popular default value of 0.75 was chosen by the original Word2Vec paper.\u001b[0m\n",
       "\u001b[0;34m            More recently, in https://arxiv.org/abs/1804.04212, Caselles-Dupré, Lesaint, & Royo-Letelier suggest that\u001b[0m\n",
       "\u001b[0;34m            other values may perform better for recommendation applications.\u001b[0m\n",
       "\u001b[0;34m        cbow_mean : {0, 1}, optional\u001b[0m\n",
       "\u001b[0;34m            If 0, use the sum of the context word vectors. If 1, use the mean, only applies when cbow is used.\u001b[0m\n",
       "\u001b[0;34m        alpha : float, optional\u001b[0m\n",
       "\u001b[0;34m            The initial learning rate.\u001b[0m\n",
       "\u001b[0;34m        min_alpha : float, optional\u001b[0m\n",
       "\u001b[0;34m            Learning rate will linearly drop to `min_alpha` as training progresses.\u001b[0m\n",
       "\u001b[0;34m        seed : int, optional\u001b[0m\n",
       "\u001b[0;34m            Seed for the random number generator. Initial vectors for each word are seeded with a hash of\u001b[0m\n",
       "\u001b[0;34m            the concatenation of word + `str(seed)`. Note that for a fully deterministically-reproducible run,\u001b[0m\n",
       "\u001b[0;34m            you must also limit the model to a single worker thread (`workers=1`), to eliminate ordering jitter\u001b[0m\n",
       "\u001b[0;34m            from OS thread scheduling. (In Python 3, reproducibility between interpreter launches also requires\u001b[0m\n",
       "\u001b[0;34m            use of the `PYTHONHASHSEED` environment variable to control hash randomization).\u001b[0m\n",
       "\u001b[0;34m        max_vocab_size : int, optional\u001b[0m\n",
       "\u001b[0;34m            Limits the RAM during vocabulary building; if there are more unique\u001b[0m\n",
       "\u001b[0;34m            words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM.\u001b[0m\n",
       "\u001b[0;34m            Set to `None` for no limit.\u001b[0m\n",
       "\u001b[0;34m        max_final_vocab : int, optional\u001b[0m\n",
       "\u001b[0;34m            Limits the vocab to a target vocab size by automatically picking a matching min_count. If the specified\u001b[0m\n",
       "\u001b[0;34m            min_count is more than the calculated min_count, the specified min_count will be used.\u001b[0m\n",
       "\u001b[0;34m            Set to `None` if not required.\u001b[0m\n",
       "\u001b[0;34m        sample : float, optional\u001b[0m\n",
       "\u001b[0;34m            The threshold for configuring which higher-frequency words are randomly downsampled,\u001b[0m\n",
       "\u001b[0;34m            useful range is (0, 1e-5).\u001b[0m\n",
       "\u001b[0;34m        hashfxn : function, optional\u001b[0m\n",
       "\u001b[0;34m            Hash function to use to randomly initialize weights, for increased training reproducibility.\u001b[0m\n",
       "\u001b[0;34m        iter : int, optional\u001b[0m\n",
       "\u001b[0;34m            Number of iterations (epochs) over the corpus.\u001b[0m\n",
       "\u001b[0;34m        trim_rule : function, optional\u001b[0m\n",
       "\u001b[0;34m            Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary,\u001b[0m\n",
       "\u001b[0;34m            be trimmed away, or handled using the default (discard if word count < min_count).\u001b[0m\n",
       "\u001b[0;34m            Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`),\u001b[0m\n",
       "\u001b[0;34m            or a callable that accepts parameters (word, count, min_count) and returns either\u001b[0m\n",
       "\u001b[0;34m            :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`.\u001b[0m\n",
       "\u001b[0;34m            The rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as part of the\u001b[0m\n",
       "\u001b[0;34m            model.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            The input parameters are of the following types:\u001b[0m\n",
       "\u001b[0;34m                * `word` (str) - the word we are examining\u001b[0m\n",
       "\u001b[0;34m                * `count` (int) - the word's frequency count in the corpus\u001b[0m\n",
       "\u001b[0;34m                * `min_count` (int) - the minimum count threshold.\u001b[0m\n",
       "\u001b[0;34m        sorted_vocab : {0, 1}, optional\u001b[0m\n",
       "\u001b[0;34m            If 1, sort the vocabulary by descending frequency before assigning word indexes.\u001b[0m\n",
       "\u001b[0;34m            See :meth:`~gensim.models.word2vec.Word2VecVocab.sort_vocab()`.\u001b[0m\n",
       "\u001b[0;34m        batch_words : int, optional\u001b[0m\n",
       "\u001b[0;34m            Target size (in words) for batches of examples passed to worker threads (and\u001b[0m\n",
       "\u001b[0;34m            thus cython routines).(Larger batches will be passed if individual\u001b[0m\n",
       "\u001b[0;34m            texts are longer than 10000 words, but the standard cython code truncates to that maximum.)\u001b[0m\n",
       "\u001b[0;34m        compute_loss: bool, optional\u001b[0m\n",
       "\u001b[0;34m            If True, computes and stores loss value which can be retrieved using\u001b[0m\n",
       "\u001b[0;34m            :meth:`~gensim.models.word2vec.Word2Vec.get_latest_training_loss`.\u001b[0m\n",
       "\u001b[0;34m        callbacks : iterable of :class:`~gensim.models.callbacks.CallbackAny2Vec`, optional\u001b[0m\n",
       "\u001b[0;34m            Sequence of callbacks to be executed at specific stages during training.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Examples\u001b[0m\n",
       "\u001b[0;34m        --------\u001b[0m\n",
       "\u001b[0;34m        Initialize and train a :class:`~gensim.models.word2vec.Word2Vec` model\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. sourcecode:: pycon\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> from gensim.models import Word2Vec\u001b[0m\n",
       "\u001b[0;34m            >>> sentences = [[\"cat\", \"say\", \"meow\"], [\"dog\", \"say\", \"woof\"]]\u001b[0m\n",
       "\u001b[0;34m            >>> model = Word2Vec(sentences, min_count=1)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_final_vocab\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmax_final_vocab\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcall_on_class_only\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWord2VecKeyedVectors\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWord2VecVocab\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmax_vocab_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_vocab_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmin_count\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msorted_vocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msorted_vocab\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mnull_word\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnull_word\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_final_vocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmax_final_vocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mns_exponent\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mns_exponent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWord2VecTrainables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvector_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhashfxn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhashfxn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mWord2Vec\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0msentences\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mworkers\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvector_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0miter\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrim_rule\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrim_rule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0malpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwindow\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mseed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnegative\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnegative\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcbow_mean\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcbow_mean\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmin_alpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcompute_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_do_train_epoch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mthread_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moffset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcython_vocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mthread_private_mem\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcur_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneu1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mthread_private_mem\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msg\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mexamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtally\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mraw_tally\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_epoch_sg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moffset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcython_vocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcur_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                                        \u001b[0mtotal_examples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneu1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mexamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtally\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mraw_tally\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_epoch_cbow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moffset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcython_vocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcur_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                                          \u001b[0mtotal_examples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneu1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mexamples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtally\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mraw_tally\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_do_train_job\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Train the model on a single batch of sentences.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        sentences : iterable of list of str\u001b[0m\n",
       "\u001b[0;34m            Corpus chunk to be used in this training batch.\u001b[0m\n",
       "\u001b[0;34m        alpha : float\u001b[0m\n",
       "\u001b[0;34m            The learning rate used in this batch.\u001b[0m\n",
       "\u001b[0;34m        inits : (np.ndarray, np.ndarray)\u001b[0m\n",
       "\u001b[0;34m            Each worker threads private work memory.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns\u001b[0m\n",
       "\u001b[0;34m        -------\u001b[0m\n",
       "\u001b[0;34m        (int, int)\u001b[0m\n",
       "\u001b[0;34m             2-tuple (effective word count after ignoring unknown words and sentence length trimming, total word count).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneu1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minits\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mtally\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msg\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mtally\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain_batch_sg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mtally\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain_batch_cbow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneu1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mtally\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_raw_word_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_clear_post_train\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Remove all L2-normalized word vectors from the model.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors_norm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_set_train_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'compute_loss'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'compute_loss'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunning_training_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m              \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mword_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m              \u001b[0mqueue_factor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreport_delay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcompute_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Update the model's neural weights from a sequence of sentences.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Notes\u001b[0m\n",
       "\u001b[0;34m        -----\u001b[0m\n",
       "\u001b[0;34m        To support linear learning-rate decay from (initial) `alpha` to `min_alpha`, and accurate\u001b[0m\n",
       "\u001b[0;34m        progress-percentage logging, either `total_examples` (count of sentences) or `total_words` (count of\u001b[0m\n",
       "\u001b[0;34m        raw words in sentences) **MUST** be provided. If `sentences` is the same corpus\u001b[0m\n",
       "\u001b[0;34m        that was provided to :meth:`~gensim.models.word2vec.Word2Vec.build_vocab` earlier,\u001b[0m\n",
       "\u001b[0;34m        you can simply use `total_examples=self.corpus_count`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Warnings\u001b[0m\n",
       "\u001b[0;34m        --------\u001b[0m\n",
       "\u001b[0;34m        To avoid common mistakes around the model's ability to do multiple training passes itself, an\u001b[0m\n",
       "\u001b[0;34m        explicit `epochs` argument **MUST** be provided. In the common and recommended case\u001b[0m\n",
       "\u001b[0;34m        where :meth:`~gensim.models.word2vec.Word2Vec.train` is only called once, you can set `epochs=self.iter`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        sentences : iterable of list of str\u001b[0m\n",
       "\u001b[0;34m            The `sentences` iterable can be simply a list of lists of tokens, but for larger corpora,\u001b[0m\n",
       "\u001b[0;34m            consider an iterable that streams the sentences directly from disk/network.\u001b[0m\n",
       "\u001b[0;34m            See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus`\u001b[0m\n",
       "\u001b[0;34m            or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples.\u001b[0m\n",
       "\u001b[0;34m            See also the `tutorial on data streaming in Python\u001b[0m\n",
       "\u001b[0;34m            <https://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/>`_.\u001b[0m\n",
       "\u001b[0;34m        corpus_file : str, optional\u001b[0m\n",
       "\u001b[0;34m            Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format.\u001b[0m\n",
       "\u001b[0;34m            You may use this argument instead of `sentences` to get performance boost. Only one of `sentences` or\u001b[0m\n",
       "\u001b[0;34m            `corpus_file` arguments need to be passed (not both of them).\u001b[0m\n",
       "\u001b[0;34m        total_examples : int\u001b[0m\n",
       "\u001b[0;34m            Count of sentences.\u001b[0m\n",
       "\u001b[0;34m        total_words : int\u001b[0m\n",
       "\u001b[0;34m            Count of raw words in sentences.\u001b[0m\n",
       "\u001b[0;34m        epochs : int\u001b[0m\n",
       "\u001b[0;34m            Number of iterations (epochs) over the corpus.\u001b[0m\n",
       "\u001b[0;34m        start_alpha : float, optional\u001b[0m\n",
       "\u001b[0;34m            Initial learning rate. If supplied, replaces the starting `alpha` from the constructor,\u001b[0m\n",
       "\u001b[0;34m            for this one call to`train()`.\u001b[0m\n",
       "\u001b[0;34m            Use only if making multiple calls to `train()`, when you want to manage the alpha learning-rate yourself\u001b[0m\n",
       "\u001b[0;34m            (not recommended).\u001b[0m\n",
       "\u001b[0;34m        end_alpha : float, optional\u001b[0m\n",
       "\u001b[0;34m            Final learning rate. Drops linearly from `start_alpha`.\u001b[0m\n",
       "\u001b[0;34m            If supplied, this replaces the final `min_alpha` from the constructor, for this one call to `train()`.\u001b[0m\n",
       "\u001b[0;34m            Use only if making multiple calls to `train()`, when you want to manage the alpha learning-rate yourself\u001b[0m\n",
       "\u001b[0;34m            (not recommended).\u001b[0m\n",
       "\u001b[0;34m        word_count : int, optional\u001b[0m\n",
       "\u001b[0;34m            Count of words already trained. Set this to 0 for the usual\u001b[0m\n",
       "\u001b[0;34m            case of training on all words in sentences.\u001b[0m\n",
       "\u001b[0;34m        queue_factor : int, optional\u001b[0m\n",
       "\u001b[0;34m            Multiplier for size of queue (number of workers * queue_factor).\u001b[0m\n",
       "\u001b[0;34m        report_delay : float, optional\u001b[0m\n",
       "\u001b[0;34m            Seconds to wait before reporting progress.\u001b[0m\n",
       "\u001b[0;34m        compute_loss: bool, optional\u001b[0m\n",
       "\u001b[0;34m            If True, computes and stores loss value which can be retrieved using\u001b[0m\n",
       "\u001b[0;34m            :meth:`~gensim.models.word2vec.Word2Vec.get_latest_training_loss`.\u001b[0m\n",
       "\u001b[0;34m        callbacks : iterable of :class:`~gensim.models.callbacks.CallbackAny2Vec`, optional\u001b[0m\n",
       "\u001b[0;34m            Sequence of callbacks to be executed at specific stages during training.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Examples\u001b[0m\n",
       "\u001b[0;34m        --------\u001b[0m\n",
       "\u001b[0;34m        .. sourcecode:: pycon\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> from gensim.models import Word2Vec\u001b[0m\n",
       "\u001b[0;34m            >>> sentences = [[\"cat\", \"say\", \"meow\"], [\"dog\", \"say\", \"woof\"]]\u001b[0m\n",
       "\u001b[0;34m            >>>\u001b[0m\n",
       "\u001b[0;34m            >>> model = Word2Vec(min_count=1)\u001b[0m\n",
       "\u001b[0;34m            >>> model.build_vocab(sentences)  # prepare the model vocabulary\u001b[0m\n",
       "\u001b[0;34m            >>> model.train(sentences, total_examples=model.corpus_count, epochs=model.iter)  # train word vectors\u001b[0m\n",
       "\u001b[0;34m            (1, 30)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mWord2Vec\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0msentences\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_examples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_words\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstart_alpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mend_alpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mword_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mword_count\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mqueue_factor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mqueue_factor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreport_delay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreport_delay\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcompute_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mscore\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_sentences\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1e6\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchunksize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mqueue_factor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreport_delay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Score the log probability for a sequence of sentences.\u001b[0m\n",
       "\u001b[0;34m        This does not change the fitted model in any way (see :meth:`~gensim.models.word2vec.Word2Vec.train` for that).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Gensim has currently only implemented score for the hierarchical softmax scheme,\u001b[0m\n",
       "\u001b[0;34m        so you need to have run word2vec with `hs=1` and `negative=0` for this to work.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Note that you should specify `total_sentences`; you'll run into problems if you ask to\u001b[0m\n",
       "\u001b[0;34m        score more than this number of sentences but it is inefficient to set the value too high.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        See the `article by Matt Taddy: \"Document Classification by Inversion of Distributed Language Representations\"\u001b[0m\n",
       "\u001b[0;34m        <https://arxiv.org/pdf/1504.07295.pdf>`_ and the\u001b[0m\n",
       "\u001b[0;34m        `gensim demo <https://github.com/piskvorky/gensim/blob/develop/docs/notebooks/deepir.ipynb>`_ for examples of\u001b[0m\n",
       "\u001b[0;34m        how to use such scores in document classification.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        sentences : iterable of list of str\u001b[0m\n",
       "\u001b[0;34m            The `sentences` iterable can be simply a list of lists of tokens, but for larger corpora,\u001b[0m\n",
       "\u001b[0;34m            consider an iterable that streams the sentences directly from disk/network.\u001b[0m\n",
       "\u001b[0;34m            See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus`\u001b[0m\n",
       "\u001b[0;34m            or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples.\u001b[0m\n",
       "\u001b[0;34m        total_sentences : int, optional\u001b[0m\n",
       "\u001b[0;34m            Count of sentences.\u001b[0m\n",
       "\u001b[0;34m        chunksize : int, optional\u001b[0m\n",
       "\u001b[0;34m            Chunksize of jobs\u001b[0m\n",
       "\u001b[0;34m        queue_factor : int, optional\u001b[0m\n",
       "\u001b[0;34m            Multiplier for size of queue (number of workers * queue_factor).\u001b[0m\n",
       "\u001b[0;34m        report_delay : float, optional\u001b[0m\n",
       "\u001b[0;34m            Seconds to wait before reporting progress.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;34m\"scoring sentences with %i workers on %i vocabulary and %i features, \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;34m\"using sg=%s hs=%s sample=%s and negative=%s\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayer1_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnegative\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"you must first build vocabulary before scoring new data\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"We have currently only implemented score for the hierarchical softmax scheme, \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"so you need to have run word2vec with hs=1 and negative=0 for this to work.\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mworker_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;34m\"\"\"Compute log probability for each sentence, lifting lists of sentences from the jobs queue.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mwork\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# for sg hs, we actually only need one memory loc (running sum)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mneu1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmatutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros_aligned\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayer1_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob_queue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mjob\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# signal to finish\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mns\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mfor\u001b[0m \u001b[0msentence_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentence\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mjob\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0msentence_id\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mtotal_sentences\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msg\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mscore\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscore_sentence_sg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentence\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwork\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mscore\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscore_sentence_cbow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentence\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwork\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneu1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0msentence_scores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0msentence_id\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscore\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mns\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mprogress_queue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mns\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# report progress\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mstart\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext_report\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefault_timer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# buffer ahead only a limited number of jobs.. this is the reason we can't simply use ThreadPool :(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mjob_queue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mQueue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmaxsize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mqueue_factor\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mprogress_queue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mQueue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmaxsize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mqueue_factor\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mworkers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mthreading\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mThread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtarget\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mworker_loop\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mthread\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mworkers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mthread\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdaemon\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m  \u001b[0;31m# make interrupting the process with ctrl+c easier\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mthread\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0msentence_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0msentence_scores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmatutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros_aligned\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal_sentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mpush_done\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mdone_jobs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mjobs_source\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrouper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchunksize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# fill jobs queue with (id, sentence) job items\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mjob_no\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjobs_source\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mjob_no\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0mtotal_sentences\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;34m\"terminating after %i sentences (set higher total_sentences if you want more).\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mtotal_sentences\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mjob_no\u001b[0m \u001b[0;34m-=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mraise\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"putting job #%i in the queue\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_no\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mjob_queue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"reached end of input; waiting to finish %i outstanding jobs\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_no\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mdone_jobs\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mworkers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mjob_queue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# give the workers heads up that they can finish -- no more work!\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mpush_done\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mwhile\u001b[0m \u001b[0mdone_jobs\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mjob_no\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpush_done\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mns\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprogress_queue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpush_done\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# only block after all jobs pushed\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0msentence_count\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mns\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mdone_jobs\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0melapsed\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefault_timer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mstart\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0melapsed\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mnext_report\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m\"PROGRESS: at %.2f%% sentences, %.0f sentences/s\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;36m100.0\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0msentence_count\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentence_count\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0melapsed\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mnext_report\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0melapsed\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mreport_delay\u001b[0m  \u001b[0;31m# don't flood log, wait report_delay seconds\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;31m# loop ended by job count; really done\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mexcept\u001b[0m \u001b[0mEmpty\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mpass\u001b[0m  \u001b[0;31m# already out of loop; continue to next push\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0melapsed\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefault_timer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mstart\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear_sims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;34m\"scoring %i sentences took %.1fs, %.0f sentences/s\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0msentence_count\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melapsed\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msentence_count\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0melapsed\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0msentence_scores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0msentence_count\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mclear_sims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Remove all L2-normalized word vectors from the model, to free up memory.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        You can recompute them later again using the :meth:`~gensim.models.word2vec.Word2Vec.init_sims` method.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors_norm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mintersect_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlockf\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0municode_errors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'strict'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Merge in an input-hidden weight matrix loaded from the original C word2vec-tool format,\u001b[0m\n",
       "\u001b[0;34m        where it intersects with the current vocabulary.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        No words are added to the existing vocabulary, but intersecting words adopt the file's weights, and\u001b[0m\n",
       "\u001b[0;34m        non-intersecting words are left alone.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        fname : str\u001b[0m\n",
       "\u001b[0;34m            The file path to load the vectors from.\u001b[0m\n",
       "\u001b[0;34m        lockf : float, optional\u001b[0m\n",
       "\u001b[0;34m            Lock-factor value to be set for any imported word-vectors; the\u001b[0m\n",
       "\u001b[0;34m            default value of 0.0 prevents further updating of the vector during subsequent\u001b[0m\n",
       "\u001b[0;34m            training. Use 1.0 to allow further training updates of merged vectors.\u001b[0m\n",
       "\u001b[0;34m        binary : bool, optional\u001b[0m\n",
       "\u001b[0;34m            If True, `fname` is in the binary word2vec C format.\u001b[0m\n",
       "\u001b[0;34m        encoding : str, optional\u001b[0m\n",
       "\u001b[0;34m            Encoding of `text` for `unicode` function (python2 only).\u001b[0m\n",
       "\u001b[0;34m        unicode_errors : str, optional\u001b[0m\n",
       "\u001b[0;34m            Error handling behaviour, used as parameter for `unicode` function (python2 only).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0moverlap_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"loading projection weights from %s\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mwith\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfin\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mheader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_unicode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mvocab_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvector_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# throws for invalid file format\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvector_size\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvector_size\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"incompatible vector size %d in file %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mvector_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# TOCONSIDER: maybe mismatched vectors still useful enough to merge (truncating/padding)?\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mbinary_len\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitemsize\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mvector_size\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvocab_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;31m# mixed text and binary: read text first, then binary\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mword\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32mif\u001b[0m \u001b[0mch\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34mb' '\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32mif\u001b[0m \u001b[0mch\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34mb'\\n'\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# ignore newlines in front of words (some binary files have)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0mword\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mword\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_unicode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mb''\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0municode_errors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mweights\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfromstring\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbinary_len\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0moverlap_count\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mweights\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors_lockf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlockf\u001b[0m  \u001b[0;31m# lock-factor: 0.0=no changes\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mfor\u001b[0m \u001b[0mline_no\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfin\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mparts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_unicode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrstrip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0municode_errors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparts\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mvector_size\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"invalid vector on line %s (is this really the text format?)\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mline_no\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mword\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweights\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparts\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mparts\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0moverlap_count\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mweights\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors_lockf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlockf\u001b[0m  \u001b[0;31m# lock-factor: 0.0=no changes\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"merged %d vectors into %s matrix from %s\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moverlap_count\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mdeprecated\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Method will be removed in 4.0.0, use self.wv.__getitem__() instead\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use `self.wv.__getitem__` instead.\u001b[0m\n",
       "\u001b[0;34m        Refer to the documentation for :meth:`~gensim.models.keyedvectors.Word2VecKeyedVectors.__getitem__`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mdeprecated\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Method will be removed in 4.0.0, use self.wv.__contains__() instead\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__contains__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mword\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use `self.wv.__contains__` instead.\u001b[0m\n",
       "\u001b[0;34m        Refer to the documentation for :meth:`~gensim.models.keyedvectors.Word2VecKeyedVectors.__contains__`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__contains__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mpredict_output_word\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontext_words_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtopn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Get the probability distribution of the center word given context words.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        context_words_list : list of str\u001b[0m\n",
       "\u001b[0;34m            List of context words.\u001b[0m\n",
       "\u001b[0;34m        topn : int, optional\u001b[0m\n",
       "\u001b[0;34m            Return `topn` words and their probabilities.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns\u001b[0m\n",
       "\u001b[0;34m        -------\u001b[0m\n",
       "\u001b[0;34m        list of (str, float)\u001b[0m\n",
       "\u001b[0;34m            `topn` length list of tuples of (word, probability).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnegative\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"We have currently only implemented predict_output_word for the negative sampling scheme, \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"so you need to have run word2vec with negative > 0 for this to work.\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'vectors'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'syn1neg'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Parameters required for predicting the output words not found.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mword_vocabs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mw\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mw\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mcontext_words_list\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mw\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mword_vocabs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"All the input context words are out-of-vocabulary for the current model.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mword2_indices\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mword_vocabs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0ml1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp_sum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mword2_indices\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mword2_indices\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcbow_mean\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0ml1\u001b[0m \u001b[0;34m/=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword2_indices\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# propagate hidden -> output and take softmax to get probabilities\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mprob_values\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mexp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msyn1neg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mprob_values\u001b[0m \u001b[0;34m/=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprob_values\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mtop_indices\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmatutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margsort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprob_values\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtopn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtopn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreverse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# returning the most probable output words with their probabilities\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex2word\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprob_values\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mindex1\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtop_indices\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0minit_sims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreplace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use `self.wv.init_sims` instead.\u001b[0m\n",
       "\u001b[0;34m        See :meth:`~gensim.models.keyedvectors.Word2VecKeyedVectors.init_sims`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mreplace\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'syn1'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msyn1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minit_sims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mreset_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mother_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Borrow shareable pre-built structures from `other_model` and reset hidden layer weights.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Structures copied are:\u001b[0m\n",
       "\u001b[0;34m            * Vocabulary\u001b[0m\n",
       "\u001b[0;34m            * Index to word mapping\u001b[0m\n",
       "\u001b[0;34m            * Cumulative frequency table (used for negative sampling)\u001b[0m\n",
       "\u001b[0;34m            * Cached corpus length\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Useful when testing multiple models on the same corpus in parallel.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        other_model : :class:`~gensim.models.word2vec.Word2Vec`\u001b[0m\n",
       "\u001b[0;34m            Another model to copy the internal structures from.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mother_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex2word\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mother_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex2word\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcum_table\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mother_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcum_table\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mother_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus_count\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset_weights\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnegative\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mstaticmethod\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mlog_accuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use `self.wv.log_accuracy` instead.\u001b[0m\n",
       "\u001b[0;34m        See :meth:`~gensim.models.word2vec.Word2VecKeyedVectors.log_accuracy`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mWord2VecKeyedVectors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog_accuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mdeprecated\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Method will be removed in 4.0.0, use self.wv.evaluate_word_analogies() instead\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mquestions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrestrict_vocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m30000\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmost_similar\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcase_insensitive\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use `self.wv.accuracy` instead.\u001b[0m\n",
       "\u001b[0;34m        See :meth:`~gensim.models.word2vec.Word2VecKeyedVectors.accuracy`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmost_similar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmost_similar\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mWord2VecKeyedVectors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmost_similar\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquestions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrestrict_vocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmost_similar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcase_insensitive\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__str__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Human readable representation of the model's state.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns\u001b[0m\n",
       "\u001b[0;34m        -------\u001b[0m\n",
       "\u001b[0;34m        str\u001b[0m\n",
       "\u001b[0;34m            Human readable representation of the model's state, including the vocabulary size, vector size\u001b[0m\n",
       "\u001b[0;34m            and learning rate.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0;34m\"%s(vocab=%s, size=%s, alpha=%s)\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex2word\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvector_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0malpha\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mdelete_temporary_training_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreplace_word_vectors_with_normalized\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Discard parameters that are used in training and scoring, to save memory.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Warnings\u001b[0m\n",
       "\u001b[0;34m        --------\u001b[0m\n",
       "\u001b[0;34m        Use only if you're sure you're done training a model.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        replace_word_vectors_with_normalized : bool, optional\u001b[0m\n",
       "\u001b[0;34m            If True, forget the original (not normalized) word vectors and only keep\u001b[0m\n",
       "\u001b[0;34m            the L2-normalized word vectors, to save even more memory.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mreplace_word_vectors_with_normalized\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minit_sims\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_minimize_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Save the model.\u001b[0m\n",
       "\u001b[0;34m        This saved model can be loaded again using :func:`~gensim.models.word2vec.Word2Vec.load`, which supports\u001b[0m\n",
       "\u001b[0;34m        online training and getting vectors for vocabulary words.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        fname : str\u001b[0m\n",
       "\u001b[0;34m            Path to the file.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# don't bother storing the cached normalized vectors, recalculable table\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'ignore'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'ignore'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'vectors_norm'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'cum_table'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mWord2Vec\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mget_latest_training_loss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Get current value of the training loss.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns\u001b[0m\n",
       "\u001b[0;34m        -------\u001b[0m\n",
       "\u001b[0;34m        float\u001b[0m\n",
       "\u001b[0;34m            Current training loss.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunning_training_loss\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mdeprecated\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"Method will be removed in 4.0.0, keep just_word_vectors = model.wv to retain just the KeyedVectors instance\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_minimize_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msave_syn1\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msave_syn1neg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msave_vectors_lockf\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0msave_syn1\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0msave_syn1neg\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0msave_vectors_lockf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'syn1'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msave_syn1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msyn1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'syn1neg'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msave_syn1neg\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msyn1neg\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'vectors_lockf'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msave_vectors_lockf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrainables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors_lockf\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_trimmed_post_training\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mclassmethod\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mload_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfvocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0municode_errors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'strict'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mlimit\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatatype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use :meth:`gensim.models.KeyedVectors.load_word2vec_format` instead.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mraise\u001b[0m \u001b[0mDeprecationWarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Deprecated. Use gensim.models.KeyedVectors.load_word2vec_format instead.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0msave_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfvocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Deprecated. Use `model.wv.save_word2vec_format` instead.\u001b[0m\n",
       "\u001b[0;34m        See :meth:`gensim.models.KeyedVectors.save_word2vec_format`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mraise\u001b[0m \u001b[0mDeprecationWarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Deprecated. Use model.wv.save_word2vec_format instead.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0mclassmethod\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"Load a previously saved :class:`~gensim.models.word2vec.Word2Vec` model.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        See Also\u001b[0m\n",
       "\u001b[0;34m        --------\u001b[0m\n",
       "\u001b[0;34m        :meth:`~gensim.models.word2vec.Word2Vec.save`\u001b[0m\n",
       "\u001b[0;34m            Save model.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Parameters\u001b[0m\n",
       "\u001b[0;34m        ----------\u001b[0m\n",
       "\u001b[0;34m        fname : str\u001b[0m\n",
       "\u001b[0;34m            Path to the saved file.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns\u001b[0m\n",
       "\u001b[0;34m        -------\u001b[0m\n",
       "\u001b[0;34m        :class:`~gensim.models.word2vec.Word2Vec`\u001b[0m\n",
       "\u001b[0;34m            Loaded model.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mWord2Vec\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;31m# for backward compatibility for `max_final_vocab` feature\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'max_final_vocab'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_final_vocab\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocabulary\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_final_vocab\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Model saved using code from earlier Gensim Version. Re-loading old model in a compatible way.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mfrom\u001b[0m \u001b[0mgensim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdeprecated\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mword2vec\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mload_old_word2vec\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mload_old_word2vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mFile:\u001b[0m           ~/.local/lib/python3.6/site-packages/gensim/models/word2vec.py\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Word2Vec??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "uuid": "dc6c110d-fa06-4d0c-b114-4d1ae4d6ac23"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00185469, -0.03886808,  0.03721781, -0.04123814,  0.02220134,\n",
       "       -0.00444706, -0.03115   , -0.02864965,  0.02521354,  0.01577712,\n",
       "        0.03325493,  0.00947807], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import Word2Vec\n",
    "doc = [['30760', '157507'],\n",
    "       ['289197', '63746'],\n",
    "       ['36162', '168401'],\n",
    "       ['50644', '36162']]\n",
    "w2v = Word2Vec(doc, size=12, sg=1, window=2, seed=2020, workers=2, min_count=1, iter=1)\n",
    "\n",
    "# 查看'30760'表示的词向量\n",
    "w2v['30760']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "uuid": "a5544a26-9777-4236-bbd4-9647c5a86071"
   },
   "outputs": [],
   "source": [
    "def trian_item_word2vec(click_df, embed_size=64, save_name='item_w2v_emb.pkl', split_char=' '):\n",
    "    click_df = click_df.sort_values('click_timestamp')\n",
    "    # 只有转换成字符串才可以进行训练\n",
    "    click_df['click_article_id'] = click_df['click_article_id'].astype(str)\n",
    "    # 转换成句子的形式\n",
    "    docs = click_df.groupby(['user_id'])['click_article_id'].apply(lambda x: list(x)).reset_index()\n",
    "    docs = docs['click_article_id'].values.tolist()\n",
    "\n",
    "    # 为了方便查看训练的进度，这里设定一个log信息\n",
    "    logging.basicConfig(format='%(asctime)s:%(levelname)s:%(message)s', level=logging.INFO)\n",
    "\n",
    "    # 这里的参数对训练得到的向量影响也很大,默认负采样为5\n",
    "    w2v = Word2Vec(docs, size=16, sg=1, window=5, seed=2020, workers=24, min_count=1, iter=1)\n",
    "    \n",
    "    # 保存成字典的形式\n",
    "    item_w2v_emb_dict = {k: w2v[k] for k in click_df['click_article_id']}\n",
    "    pickle.dump(item_w2v_emb_dict, open(save_path + 'item_w2v_emb.pkl', 'wb'))\n",
    "    \n",
    "    return item_w2v_emb_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "uuid": "83b1daf2-020e-444b-a772-7a005750da8d"
   },
   "outputs": [],
   "source": [
    "# 可以通过字典查询对应的item的Embedding\n",
    "def get_embedding(save_path, all_click_df):\n",
    "    if os.path.exists(save_path + 'item_content_emb.pkl'):\n",
    "        item_content_emb_dict = pickle.load(open(save_path + 'item_content_emb.pkl', 'rb'))\n",
    "    else:\n",
    "        print('item_content_emb.pkl 文件不存在...')\n",
    "        \n",
    "    # w2v Embedding是需要提前训练好的\n",
    "    if os.path.exists(save_path + 'item_w2v_emb.pkl'):\n",
    "        item_w2v_emb_dict = pickle.load(open(save_path + 'item_w2v_emb.pkl', 'rb'))\n",
    "    else:\n",
    "        item_w2v_emb_dict = trian_item_word2vec(all_click_df)\n",
    "        \n",
    "    if os.path.exists(save_path + 'item_youtube_emb.pkl'):\n",
    "        item_youtube_emb_dict = pickle.load(open(save_path + 'item_youtube_emb.pkl', 'rb'))\n",
    "    else:\n",
    "        print('item_youtube_emb.pkl 文件不存在...')\n",
    "    \n",
    "    if os.path.exists(save_path + 'user_youtube_emb.pkl'):\n",
    "        user_youtube_emb_dict = pickle.load(open(save_path + 'user_youtube_emb.pkl', 'rb'))\n",
    "    else:\n",
    "        print('user_youtube_emb.pkl 文件不存在...')\n",
    "    \n",
    "    return item_content_emb_dict, item_w2v_emb_dict, item_youtube_emb_dict, user_youtube_emb_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "44311977-6351-4184-8aa0-d7faa99ac3a2"
   },
   "source": [
    "#### 读取文章信息 读取数据\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "uuid": "beaf0653-8a06-49db-9f7d-f94ab6336042"
   },
   "outputs": [],
   "source": [
    "def get_article_info_df():\n",
    "    article_info_df = pd.read_csv(data_path + 'articles.csv')\n",
    "    article_info_df = reduce_mem(article_info_df)\n",
    "    \n",
    "    return article_info_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "uuid": "62cf4f29-1868-45b7-9bdd-4784be213106"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- Mem. usage decreased to 23.34 Mb (69.4% reduction),time spend:0.00 min\n"
     ]
    }
   ],
   "source": [
    "# 这里offline的online的区别就是验证集是否为空\n",
    "click_trn, click_val, click_tst, val_ans = get_trn_val_tst_data(data_path, offline=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "uuid": "b18aaef0-02f5-4ea7-a1a0-8f9049f68c50"
   },
   "outputs": [],
   "source": [
    "click_trn_hist, click_trn_last = get_hist_and_last_click(click_trn)\n",
    "\n",
    "if click_val is not None:\n",
    "    click_val_hist, click_val_last = click_val, val_ans\n",
    "else:\n",
    "    click_val_hist, click_val_last = None, None\n",
    "    \n",
    "click_tst_hist = click_tst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "7123cd2b-af1f-425e-b38f-0470ad581e79"
   },
   "source": [
    "### 对训练数据做负采样\n",
    "通过召回我们将数据转换成三元组的形式（user1, item1, label）的形式，观察发现正负样本差距极度不平衡，我们可以先对负样本进行下采样，下采样的目的一方面缓解了正负样本比例的问题，另一方面也减小了我们做排序特征的压力，我们在做负采样的时候又有哪些东西是需要注意的呢？\n",
    "\n",
    "只对负样本进行下采样(如果有比较好的正样本扩充的方法其实也是可以考虑的)  \n",
    "负采样之后，保证所有的用户和文章仍然出现在采样之后的数据中  \n",
    "下采样的比例可以根据实际情况人为的控制  \n",
    "做完负采样之后，更新此时新的用户召回文章列表，因为后续做特征的时候可能用到相对位置的信息。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "aa2f37d3-abcb-4244-a95a-d401e8c66057"
   },
   "outputs": [],
   "source": [
    "# 将召回列表转换成df的形式\n",
    "def recall_dict_2_df(recall_list_dict):\n",
    "    df_row_list = [] # [user, item, score]\n",
    "    for user, recall_list in tqdm(recall_list_dict.items()):\n",
    "        for item, score in recall_list:\n",
    "            df_row_list.append([user, item, score])\n",
    "    \n",
    "    col_names = ['user_id', 'sim_item', 'score']\n",
    "    recall_list_df = pd.DataFrame(df_row_list, columns=col_names)\n",
    "    \n",
    "    return recall_list_df\n",
    "# 负采样函数，这里可以控制负采样时的比例, 这里给了一个默认的值\n",
    "def neg_sample_recall_data(recall_items_df, sample_rate=0.001):\n",
    "    pos_data = recall_items_df[recall_items_df['label'] == 1]\n",
    "    neg_data = recall_items_df[recall_items_df['label'] == 0]\n",
    "    \n",
    "    print('pos_data_num:', len(pos_data), 'neg_data_num:', len(neg_data), 'pos/neg:', len(pos_data)/len(neg_data))\n",
    "    \n",
    "    # 分组采样函数\n",
    "    def neg_sample_func(group_df):\n",
    "        neg_num = len(group_df)\n",
    "        sample_num = max(int(neg_num * sample_rate), 1) # 保证最少有一个\n",
    "        sample_num = min(sample_num, 5) # 保证最多不超过5个，这里可以根据实际情况进行选择\n",
    "        return group_df.sample(n=sample_num, replace=True)\n",
    "    \n",
    "    # 对用户进行负采样，保证所有用户都在采样后的数据中\n",
    "    neg_data_user_sample = neg_data.groupby('user_id', group_keys=False).apply(neg_sample_func)\n",
    "    # 对文章进行负采样，保证所有文章都在采样后的数据中\n",
    "    neg_data_item_sample = neg_data.groupby('sim_item', group_keys=False).apply(neg_sample_func)\n",
    "    \n",
    "    # 将上述两种情况下的采样数据合并\n",
    "    neg_data_new = neg_data_user_sample.append(neg_data_item_sample)\n",
    "    # 由于上述两个操作是分开的，可能将两个相同的数据给重复选择了，所以需要对合并后的数据进行去重\n",
    "    neg_data_new = neg_data_new.sort_values(['user_id', 'score']).drop_duplicates(['user_id', 'sim_item'], keep='last')\n",
    "    \n",
    "    # 将正样本数据合并\n",
    "    data_new = pd.concat([pos_data, neg_data_new], ignore_index=True)\n",
    "    \n",
    "    return data_new\n",
    "# 召回数据打标签\n",
    "def get_rank_label_df(recall_list_df, label_df, is_test=False):\n",
    "    # 测试集是没有标签了，为了后面代码同一一些，这里直接给一个负数替代\n",
    "    if is_test:\n",
    "        recall_list_df['label'] = -1\n",
    "        return recall_list_df\n",
    "    \n",
    "    label_df = label_df.rename(columns={'click_article_id': 'sim_item'})\n",
    "    recall_list_df_ = recall_list_df.merge(label_df[['user_id', 'sim_item', 'click_timestamp']], \\\n",
    "                                               how='left', on=['user_id', 'sim_item'])\n",
    "    recall_list_df_['label'] = recall_list_df_['click_timestamp'].apply(lambda x: 0.0 if np.isnan(x) else 1.0)\n",
    "    del recall_list_df_['click_timestamp']\n",
    "    \n",
    "    return recall_list_df_\n",
    "def get_user_recall_item_label_df(click_trn_hist, click_val_hist, click_tst_hist,click_trn_last, click_val_last, recall_list_df):\n",
    "    # 获取训练数据的召回列表\n",
    "    trn_user_items_df = recall_list_df[recall_list_df['user_id'].isin(click_trn_hist['user_id'].unique())]\n",
    "    # 训练数据打标签\n",
    "    trn_user_item_label_df = get_rank_label_df(trn_user_items_df, click_trn_last, is_test=False)\n",
    "    # 训练数据负采样\n",
    "    trn_user_item_label_df = neg_sample_recall_data(trn_user_item_label_df)\n",
    "    \n",
    "    if click_val is not None:\n",
    "        val_user_items_df = recall_list_df[recall_list_df['user_id'].isin(click_val_hist['user_id'].unique())]\n",
    "        val_user_item_label_df = get_rank_label_df(val_user_items_df, click_val_last, is_test=False)\n",
    "        val_user_item_label_df = neg_sample_recall_data(val_user_item_label_df)\n",
    "    else:\n",
    "        val_user_item_label_df = None\n",
    "        \n",
    "    # 测试数据不需要进行负采样，直接对所有的召回商品进行打-1标签\n",
    "    tst_user_items_df = recall_list_df[recall_list_df['user_id'].isin(click_tst_hist['user_id'].unique())]\n",
    "    tst_user_item_label_df = get_rank_label_df(tst_user_items_df, None, is_test=True)\n",
    "    \n",
    "    return trn_user_item_label_df, val_user_item_label_df, tst_user_item_label_df\n",
    "# 读取召回列表\n",
    "recall_list_dict = get_recall_list(save_path, single_recall_model='i2i_itemcf') # 这里只选择了单路召回的结果，也可以选择多路召回结果\n",
    "# 将召回数据转换成df\n",
    "recall_list_df = recall_dict_2_df(recall_list_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "b54cd57e-1dc4-48f9-bf0f-2156a2c99155"
   },
   "outputs": [],
   "source": [
    "# 给训练验证数据打标签，并负采样（这一部分时间比较久）\n",
    "trn_user_item_label_df, val_user_item_label_df, tst_user_item_label_df = get_user_recall_item_label_df(click_trn_hist, \n",
    "                                                                                                       click_val_hist, \n",
    "                                                                                                       click_tst_hist,\n",
    "                                                                                                       click_trn_last, \n",
    "                                                                                                       click_val_last, \n",
    "                                                                                                       recall_list_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "0487f500-c334-4295-afdf-41f9b43983f2"
   },
   "source": [
    "#### 将召回数据转换成字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "uuid": "8de9a8b3-61a6-4dda-a578-524b74762d81"
   },
   "outputs": [],
   "source": [
    "# 将最终的召回的df数据转换成字典的形式做排序特征\n",
    "def make_tuple_func(group_df):\n",
    "    row_data = []\n",
    "    for name, row_df in group_df.iterrows():\n",
    "        row_data.append((row_df['sim_item'], row_df['score'], row_df['label']))\n",
    "    \n",
    "    return row_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "b6cc5326-534b-4d0e-aa6b-48a5d9a41ea8"
   },
   "outputs": [],
   "source": [
    "trn_user_item_label_tuples = trn_user_item_label_df.groupby('user_id').apply(make_tuple_func).reset_index()\n",
    "trn_user_item_label_tuples_dict = dict(zip(trn_user_item_label_tuples['user_id'], trn_user_item_label_tuples[0]))\n",
    "\n",
    "if val_user_item_label_df is not None:\n",
    "    val_user_item_label_tuples = val_user_item_label_df.groupby('user_id').apply(make_tuple_func).reset_index()\n",
    "    val_user_item_label_tuples_dict = dict(zip(val_user_item_label_tuples['user_id'], val_user_item_label_tuples[0]))\n",
    "else:\n",
    "    val_user_item_label_tuples_dict = None\n",
    "    \n",
    "tst_user_item_label_tuples = tst_user_item_label_df.groupby('user_id').apply(make_tuple_func).reset_index()\n",
    "tst_user_item_label_tuples_dict = dict(zip(tst_user_item_label_tuples['user_id'], tst_user_item_label_tuples[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "113d27c6-4547-4989-a40d-1dae27245d4c"
   },
   "source": [
    "### 用户历史行为相关特征\n",
    "对于每个用户召回的每个商品， 做特征。 具体步骤如下：\n",
    "\n",
    "对于每个用户， 获取最后点击的N个商品的item_id，\n",
    "对于该用户的每个召回商品， 计算与上面最后N次点击商品的相似度的和(最大， 最小，均值)， 时间差特征，相似性特征，字数差特征，与该用户的相似性特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "uuid": "10798982-3dc0-4d76-84d0-0550d51a6ded"
   },
   "outputs": [],
   "source": [
    "# 下面基于data做历史相关的特征\n",
    "def create_feature(users_id, recall_list, click_hist_df,  articles_info, articles_emb, user_emb=None, N=1):\n",
    "    \"\"\"\n",
    "    基于用户的历史行为做相关特征\n",
    "    :param users_id: 用户id\n",
    "    :param recall_list: 对于每个用户召回的候选文章列表\n",
    "    :param click_hist_df: 用户的历史点击信息\n",
    "    :param articles_info: 文章信息\n",
    "    :param articles_emb: 文章的embedding向量, 这个可以用item_content_emb, item_w2v_emb, item_youtube_emb\n",
    "    :param user_emb: 用户的embedding向量， 这个是user_youtube_emb, 如果没有也可以不用， 但要注意如果要用的话， articles_emb就要用item_youtube_emb的形式， 这样维度才一样\n",
    "    :param N: 最近的N次点击  由于testA日志里面很多用户只存在一次历史点击， 所以为了不产生空值，默认是1\n",
    "    \"\"\"\n",
    "    \n",
    "    # 建立一个二维列表保存结果， 后面要转成DataFrame\n",
    "    all_user_feas = []\n",
    "    i = 0\n",
    "    for user_id in tqdm(users_id):\n",
    "        # 该用户的最后N次点击\n",
    "        hist_user_items = click_hist_df[click_hist_df['user_id']==user_id]['click_article_id'][-N:]\n",
    "        \n",
    "        # 遍历该用户的召回列表\n",
    "        for rank, (article_id, score, label) in enumerate(recall_list[user_id]):\n",
    "            # 该文章建立时间, 字数\n",
    "            a_create_time = articles_info[articles_info['article_id']==article_id]['created_at_ts'].values[0]\n",
    "            a_words_count = articles_info[articles_info['article_id']==article_id]['words_count'].values[0]\n",
    "            single_user_fea = [user_id, article_id]\n",
    "            # 计算与最后点击的商品的相似度的和， 最大值和最小值， 均值\n",
    "            sim_fea = []\n",
    "            time_fea = []\n",
    "            word_fea = []\n",
    "            # 遍历用户的最后N次点击文章\n",
    "            for hist_item in hist_user_items:\n",
    "                b_create_time = articles_info[articles_info['article_id']==hist_item]['created_at_ts'].values[0]\n",
    "                b_words_count = articles_info[articles_info['article_id']==hist_item]['words_count'].values[0]\n",
    "                \n",
    "                sim_fea.append(np.dot(articles_emb[hist_item], articles_emb[article_id]))\n",
    "                time_fea.append(abs(a_create_time-b_create_time))\n",
    "                word_fea.append(abs(a_words_count-b_words_count))\n",
    "                \n",
    "            single_user_fea.extend(sim_fea)      # 相似性特征\n",
    "            single_user_fea.extend(time_fea)    # 时间差特征\n",
    "            single_user_fea.extend(word_fea)    # 字数差特征\n",
    "            single_user_fea.extend([max(sim_fea), min(sim_fea), sum(sim_fea), sum(sim_fea) / len(sim_fea)])  # 相似性的统计特征\n",
    "            \n",
    "            if user_emb:  # 如果用户向量有的话， 这里计算该召回文章与用户的相似性特征 \n",
    "                single_user_fea.append(np.dot(user_emb[user_id], articles_emb[article_id]))\n",
    "                \n",
    "            single_user_fea.extend([score, rank, label])    \n",
    "            # 加入到总的表中\n",
    "            all_user_feas.append(single_user_fea)\n",
    "    \n",
    "    # 定义列名\n",
    "    id_cols = ['user_id', 'click_article_id']\n",
    "    sim_cols = ['sim' + str(i) for i in range(N)]\n",
    "    time_cols = ['time_diff' + str(i) for i in range(N)]\n",
    "    word_cols = ['word_diff' + str(i) for i in range(N)]\n",
    "    sat_cols = ['sim_max', 'sim_min', 'sim_sum', 'sim_mean']\n",
    "    user_item_sim_cols = ['user_item_sim'] if user_emb else []\n",
    "    user_score_rank_label = ['score', 'rank', 'label']\n",
    "    cols = id_cols + sim_cols + time_cols + word_cols + sat_cols + user_item_sim_cols + user_score_rank_label\n",
    "            \n",
    "    # 转成DataFrame\n",
    "    df = pd.DataFrame( all_user_feas, columns=cols)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "uuid": "12211ad2-21f2-4bcd-94d7-0697929418a0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-12-03 18:21:19,039:INFO:collecting all words and their counts\n",
      "2020-12-03 18:21:19,040:INFO:PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
      "2020-12-03 18:21:19,054:INFO:PROGRESS: at sentence #10000, processed 25727 words, keeping 3473 word types\n",
      "2020-12-03 18:21:19,068:INFO:PROGRESS: at sentence #20000, processed 53883 words, keeping 5811 word types\n",
      "2020-12-03 18:21:19,081:INFO:PROGRESS: at sentence #30000, processed 84881 words, keeping 7676 word types\n",
      "2020-12-03 18:21:19,097:INFO:PROGRESS: at sentence #40000, processed 118390 words, keeping 9297 word types\n",
      "2020-12-03 18:21:19,114:INFO:PROGRESS: at sentence #50000, processed 154179 words, keeping 10844 word types\n",
      "2020-12-03 18:21:19,132:INFO:PROGRESS: at sentence #60000, processed 192350 words, keeping 12357 word types\n",
      "2020-12-03 18:21:19,149:INFO:PROGRESS: at sentence #70000, processed 233685 words, keeping 13473 word types\n",
      "2020-12-03 18:21:19,169:INFO:PROGRESS: at sentence #80000, processed 281335 words, keeping 14939 word types\n",
      "2020-12-03 18:21:19,189:INFO:PROGRESS: at sentence #90000, processed 329973 words, keeping 16420 word types\n",
      "2020-12-03 18:21:19,210:INFO:PROGRESS: at sentence #100000, processed 379428 words, keeping 17904 word types\n",
      "2020-12-03 18:21:19,238:INFO:PROGRESS: at sentence #110000, processed 431464 words, keeping 18928 word types\n",
      "2020-12-03 18:21:19,262:INFO:PROGRESS: at sentence #120000, processed 489655 words, keeping 20157 word types\n",
      "2020-12-03 18:21:19,286:INFO:PROGRESS: at sentence #130000, processed 550375 words, keeping 21588 word types\n",
      "2020-12-03 18:21:19,311:INFO:PROGRESS: at sentence #140000, processed 613031 words, keeping 22923 word types\n",
      "2020-12-03 18:21:19,337:INFO:PROGRESS: at sentence #150000, processed 678645 words, keeping 24209 word types\n",
      "2020-12-03 18:21:19,364:INFO:PROGRESS: at sentence #160000, processed 749559 words, keeping 25743 word types\n",
      "2020-12-03 18:21:19,394:INFO:PROGRESS: at sentence #170000, processed 831064 words, keeping 27232 word types\n",
      "2020-12-03 18:21:19,425:INFO:PROGRESS: at sentence #180000, processed 914233 words, keeping 28612 word types\n",
      "2020-12-03 18:21:19,458:INFO:PROGRESS: at sentence #190000, processed 1004976 words, keeping 29699 word types\n",
      "2020-12-03 18:21:19,497:INFO:PROGRESS: at sentence #200000, processed 1112623 words, keeping 31116 word types\n",
      "2020-12-03 18:21:19,529:INFO:PROGRESS: at sentence #210000, processed 1200577 words, keeping 31798 word types\n",
      "2020-12-03 18:21:19,560:INFO:PROGRESS: at sentence #220000, processed 1285942 words, keeping 32381 word types\n",
      "2020-12-03 18:21:19,595:INFO:PROGRESS: at sentence #230000, processed 1380836 words, keeping 33131 word types\n",
      "2020-12-03 18:21:19,638:INFO:PROGRESS: at sentence #240000, processed 1498710 words, keeping 34213 word types\n",
      "2020-12-03 18:21:19,684:INFO:collected 35380 word types from a corpus of 1630633 raw words and 250000 sentences\n",
      "2020-12-03 18:21:19,685:INFO:Loading a fresh vocabulary\n",
      "2020-12-03 18:21:19,958:INFO:effective_min_count=1 retains 35380 unique words (100% of original 35380, drops 0)\n",
      "2020-12-03 18:21:19,959:INFO:effective_min_count=1 leaves 1630633 word corpus (100% of original 1630633, drops 0)\n",
      "2020-12-03 18:21:20,088:INFO:deleting the raw counts dictionary of 35380 items\n",
      "2020-12-03 18:21:20,089:INFO:sample=0.001 downsamples 73 most-common words\n",
      "2020-12-03 18:21:20,090:INFO:downsampling leaves estimated 1452855 word corpus (89.1% of prior 1630633)\n",
      "2020-12-03 18:21:20,181:INFO:estimated required memory for 35380 words and 16 dimensions: 22218640 bytes\n",
      "2020-12-03 18:21:20,182:INFO:resetting layer weights\n",
      "2020-12-03 18:21:20,552:INFO:training model with 24 workers on 35380 vocabulary and 16 features, using sg=1 hs=0 sample=0.001 negative=5 window=5\n",
      "2020-12-03 18:21:21,577:INFO:EPOCH 1 - PROGRESS: at 75.22% examples, 882599 words/s, in_qsize 46, out_qsize 1\n",
      "2020-12-03 18:21:21,838:INFO:worker thread finished; awaiting finish of 23 more threads\n",
      "2020-12-03 18:21:21,858:INFO:worker thread finished; awaiting finish of 22 more threads\n",
      "2020-12-03 18:21:21,871:INFO:worker thread finished; awaiting finish of 21 more threads\n",
      "2020-12-03 18:21:21,872:INFO:worker thread finished; awaiting finish of 20 more threads\n",
      "2020-12-03 18:21:21,882:INFO:worker thread finished; awaiting finish of 19 more threads\n",
      "2020-12-03 18:21:21,914:INFO:worker thread finished; awaiting finish of 18 more threads\n",
      "2020-12-03 18:21:21,915:INFO:worker thread finished; awaiting finish of 17 more threads\n",
      "2020-12-03 18:21:21,915:INFO:worker thread finished; awaiting finish of 16 more threads\n",
      "2020-12-03 18:21:21,923:INFO:worker thread finished; awaiting finish of 15 more threads\n",
      "2020-12-03 18:21:21,928:INFO:worker thread finished; awaiting finish of 14 more threads\n",
      "2020-12-03 18:21:21,931:INFO:worker thread finished; awaiting finish of 13 more threads\n",
      "2020-12-03 18:21:21,939:INFO:worker thread finished; awaiting finish of 12 more threads\n",
      "2020-12-03 18:21:21,946:INFO:worker thread finished; awaiting finish of 11 more threads\n",
      "2020-12-03 18:21:21,948:INFO:worker thread finished; awaiting finish of 10 more threads\n",
      "2020-12-03 18:21:21,951:INFO:worker thread finished; awaiting finish of 9 more threads\n",
      "2020-12-03 18:21:21,953:INFO:worker thread finished; awaiting finish of 8 more threads\n",
      "2020-12-03 18:21:21,955:INFO:worker thread finished; awaiting finish of 7 more threads\n",
      "2020-12-03 18:21:21,956:INFO:worker thread finished; awaiting finish of 6 more threads\n",
      "2020-12-03 18:21:21,961:INFO:worker thread finished; awaiting finish of 5 more threads\n",
      "2020-12-03 18:21:21,969:INFO:worker thread finished; awaiting finish of 4 more threads\n",
      "2020-12-03 18:21:21,971:INFO:worker thread finished; awaiting finish of 3 more threads\n",
      "2020-12-03 18:21:21,977:INFO:worker thread finished; awaiting finish of 2 more threads\n",
      "2020-12-03 18:21:21,983:INFO:worker thread finished; awaiting finish of 1 more threads\n",
      "2020-12-03 18:21:21,984:INFO:worker thread finished; awaiting finish of 0 more threads\n",
      "2020-12-03 18:21:21,985:INFO:EPOCH - 1 : training on 1630633 raw words (1452620 effective words) took 1.4s, 1026363 effective words/s\n",
      "2020-12-03 18:21:21,985:INFO:training on a 1630633 raw words (1452620 effective words) took 1.4s, 1014271 effective words/s\n",
      "2020-12-03 18:21:21,986:WARNING:under 10 jobs per worker: consider setting a smaller `batch_words' for smoother alpha decay\n"
     ]
    },
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: './temp_results/item_w2v_emb.pkl'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-4c68d7511b01>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0marticle_info_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_article_info_df\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mall_click\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclick_trn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclick_tst\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mitem_content_emb_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mitem_w2v_emb_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mitem_youtube_emb_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muser_youtube_emb_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_embedding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msave_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mall_click\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-14-27da4d04ba6e>\u001b[0m in \u001b[0;36mget_embedding\u001b[0;34m(save_path, all_click_df)\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0mitem_w2v_emb_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpickle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msave_path\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'item_w2v_emb.pkl'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m         \u001b[0mitem_w2v_emb_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrian_item_word2vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mall_click_df\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msave_path\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'item_youtube_emb.pkl'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-13-f2ac82cb999d>\u001b[0m in \u001b[0;36mtrian_item_word2vec\u001b[0;34m(click_df, embed_size, save_name, split_char)\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0;31m# 保存成字典的形式\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m     \u001b[0mitem_w2v_emb_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mw2v\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mclick_df\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'click_article_id'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m     \u001b[0mpickle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem_w2v_emb_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msave_path\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'item_w2v_emb.pkl'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'wb'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mitem_w2v_emb_dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: './temp_results/item_w2v_emb.pkl'"
     ]
    }
   ],
   "source": [
    "article_info_df = get_article_info_df()\n",
    "all_click = click_trn.append(click_tst)\n",
    "item_content_emb_dict, item_w2v_emb_dict, item_youtube_emb_dict, user_youtube_emb_dict = get_embedding(save_path, all_click)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "b16602cd-083e-426d-b824-3d49ae53992b"
   },
   "outputs": [],
   "source": [
    "# 获取训练验证及测试数据中召回列文章相关特征\n",
    "trn_user_item_feats_df = create_feature(trn_user_item_label_tuples_dict.keys(), trn_user_item_label_tuples_dict, \\\n",
    "                                            click_trn_hist, article_info_df, item_content_emb_dict)\n",
    "\n",
    "if val_user_item_label_tuples_dict is not None:\n",
    "    val_user_item_feats_df = create_feature(val_user_item_label_tuples_dict.keys(), val_user_item_label_tuples_dict, \\\n",
    "                                                click_val_hist, article_info_df, item_content_emb_dict)\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "    \n",
    "tst_user_item_feats_df = create_feature(tst_user_item_label_tuples_dict.keys(), tst_user_item_label_tuples_dict, \\\n",
    "                                            click_tst_hist, article_info_df, item_content_emb_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "80cdb1cf-d5f2-4b68-bd86-feac3d0c5744"
   },
   "outputs": [],
   "source": [
    "# 保存一份省的每次都要重新跑，每次跑的时间都比较长\n",
    "trn_user_item_feats_df.to_csv(save_path + 'trn_user_item_feats_df.csv', index=False)\n",
    "\n",
    "if val_user_item_feats_df is not None:\n",
    "    val_user_item_feats_df.to_csv(save_path + 'val_user_item_feats_df.csv', index=False)\n",
    "\n",
    "tst_user_item_feats_df.to_csv(save_path + 'tst_user_item_feats_df.csv', index=False)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "4eaf3451-a290-4991-8bbf-859a0e26ec84"
   },
   "source": [
    "### 用户和文章特征\n",
    "##### 用户相关特征\n",
    "这一块，正式进行特征工程，既要拼接上已有的特征， 也会做更多的特征出来，我们来梳理一下已有的特征和可构造特征：\n",
    "\n",
    "文章自身的特征， 文章字数，文章创建时间， 文章的embedding （articles表中)\n",
    "用户点击环境特征， 那些设备的特征(这个在df中)\n",
    "对于用户和商品还可以构造的特征：\n",
    "基于用户的点击文章次数和点击时间构造可以表现用户活跃度的特征\n",
    "基于文章被点击次数和时间构造可以反映文章热度的特征\n",
    "用户的时间统计特征： 根据其点击的历史文章列表的点击时间和文章的创建时间做统计特征，比如求均值， 这个可以反映用户对于文章时效的偏好\n",
    "用户的主题爱好特征， 对于用户点击的历史文章主题进行一个统计， 然后对于当前文章看看是否属于用户已经点击过的主题\n",
    "用户的字数爱好特征， 对于用户点击的历史文章的字数统计， 求一个均值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "uuid": "a0f4cf5b-2cef-4527-b237-d7929e1f4dae"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>click_article_id</th>\n",
       "      <th>click_timestamp</th>\n",
       "      <th>click_environment</th>\n",
       "      <th>click_deviceGroup</th>\n",
       "      <th>click_os</th>\n",
       "      <th>click_country</th>\n",
       "      <th>click_region</th>\n",
       "      <th>click_referrer_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>249999</td>\n",
       "      <td>160974</td>\n",
       "      <td>1506959142820</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>249999</td>\n",
       "      <td>160417</td>\n",
       "      <td>1506959172820</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>249998</td>\n",
       "      <td>160974</td>\n",
       "      <td>1506959056066</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>249998</td>\n",
       "      <td>202557</td>\n",
       "      <td>1506959086066</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>249997</td>\n",
       "      <td>183665</td>\n",
       "      <td>1506959088613</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  click_article_id  click_timestamp  click_environment  \\\n",
       "0   249999            160974    1506959142820                  4   \n",
       "1   249999            160417    1506959172820                  4   \n",
       "2   249998            160974    1506959056066                  4   \n",
       "3   249998            202557    1506959086066                  4   \n",
       "4   249997            183665    1506959088613                  4   \n",
       "\n",
       "   click_deviceGroup  click_os  click_country  click_region  \\\n",
       "0                  1        17              1            13   \n",
       "1                  1        17              1            13   \n",
       "2                  1        12              1            13   \n",
       "3                  1        12              1            13   \n",
       "4                  1        17              1            15   \n",
       "\n",
       "   click_referrer_type  \n",
       "0                    2  \n",
       "1                    2  \n",
       "2                    2  \n",
       "3                    2  \n",
       "4                    5  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "click_tst.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "uuid": "83894e6b-6918-4eda-9f3c-40e43884b67f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- Mem. usage decreased to  5.56 Mb (50.0% reduction),time spend:0.00 min\n",
      "-- Mem. usage decreased to 46.65 Mb (62.5% reduction),time spend:0.00 min\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1630633, 13)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取文章特征\n",
    "articles =  pd.read_csv(data_path+'articles.csv')\n",
    "articles = reduce_mem(articles)\n",
    "\n",
    "# 日志数据，就是前面的所有数据\n",
    "if click_val is not None:\n",
    "    all_data = click_trn.append(click_val)\n",
    "all_data = click_trn.append(click_tst)\n",
    "all_data = reduce_mem(all_data)\n",
    "\n",
    "# 拼上文章信息\n",
    "all_data = all_data.merge(articles, left_on='click_article_id', right_on='article_id')\n",
    "all_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "5742293e-ce9c-4f2f-9ccf-2f5a52e4488f"
   },
   "source": [
    "#### 分析一下点击时间和点击文章的次数，区分用户活跃度\n",
    "如果某个用户点击文章之间的时间间隔比较小， 同时点击的文章次数很多的话， 那么我们认为这种用户一般就是活跃用户, 当然衡量用户活跃度的方式可能多种多样， 这里我们只提供其中一种，我们写一个函数， 得到可以衡量用户活跃度的特征，逻辑如下：\n",
    "\n",
    "首先根据用户user_id分组， 对于每个用户，计算点击文章的次数， 两两点击文章时间间隔的均值\n",
    "把点击次数取倒数和时间间隔的均值统一归一化，然后两者相加合并，该值越小， 说明用户越活跃\n",
    "注意， 上面两两点击文章的时间间隔均值， 会出现如果用户只点击了一次的情况，这时候时间间隔均值那里会出现空值， 对于这种情况最后特征那里给个大数进行区分\n",
    "这个的衡量标准就是先把点击的次数取到数然后归一化， 然后点击的时间差归一化， 然后两者相加进行合并， 该值越小， 说明被点击的次数越多， 且间隔时间短。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "uuid": "7182d345-2389-4474-a0b1-6492599147bf"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>click_size</th>\n",
       "      <th>time_diff_mean</th>\n",
       "      <th>active_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.499466</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>0.499515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.499466</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>0.499515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.499466</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>0.499515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0.499466</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>0.499515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0.499466</td>\n",
       "      <td>0.000048</td>\n",
       "      <td>0.499515</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id click_size  time_diff_mean active_level\n",
       "0        0   0.499466        0.000048     0.499515\n",
       "1        1   0.499466        0.000048     0.499515\n",
       "2        2   0.499466        0.000048     0.499515\n",
       "3        3   0.499466        0.000048     0.499515\n",
       "4        4   0.499466        0.000048     0.499515"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " def active_level(all_data, cols):\n",
    "    \"\"\"\n",
    "    制作区分用户活跃度的特征\n",
    "    :param all_data: 数据集\n",
    "    :param cols: 用到的特征列\n",
    "    \"\"\"\n",
    "    data = all_data[cols]\n",
    "    data.sort_values(['user_id', 'click_timestamp'], inplace=True)\n",
    "    user_act = pd.DataFrame(data.groupby('user_id', as_index=False)[['click_article_id', 'click_timestamp']].\\\n",
    "                            agg({'click_article_id':np.size, 'click_timestamp': {list}}).values, columns=['user_id', 'click_size', 'click_timestamp'])\n",
    "    \n",
    "    # 计算时间间隔的均值\n",
    "    def time_diff_mean(l):\n",
    "        if len(l) == 1:\n",
    "            return 1\n",
    "        else:\n",
    "            return np.mean([j-i for i, j in list(zip(l[:-1], l[1:]))])\n",
    "        \n",
    "    user_act['time_diff_mean'] = user_act['click_timestamp'].apply(lambda x: time_diff_mean(x))\n",
    "    \n",
    "    # 点击次数取倒数\n",
    "    user_act['click_size'] = 1 / user_act['click_size']\n",
    "    \n",
    "    # 两者归一化\n",
    "    user_act['click_size'] = (user_act['click_size'] - user_act['click_size'].min()) / (user_act['click_size'].max() - user_act['click_size'].min())\n",
    "    user_act['time_diff_mean'] = (user_act['time_diff_mean'] - user_act['time_diff_mean'].min()) / (user_act['time_diff_mean'].max() - user_act['time_diff_mean'].min())     \n",
    "    user_act['active_level'] = user_act['click_size'] + user_act['time_diff_mean']\n",
    "    \n",
    "    user_act['user_id'] = user_act['user_id'].astype('int')\n",
    "    del user_act['click_timestamp']\n",
    "    \n",
    "    return user_act\n",
    "user_act_fea = active_level(all_data, ['user_id', 'click_article_id', 'click_timestamp'])\n",
    "user_act_fea.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "8e655adf-fc51-4e10-b493-2a4aea4dd948"
   },
   "source": [
    "#### 分析一下点击时间和被点击文章的次数， 衡量文章热度特征\n",
    "和上面同样的思路， 如果一篇文章在很短的时间间隔之内被点击了很多次， 说明文章比较热门，实现的逻辑和上面的基本一致， 只不过这里是按照点击的文章进行分组：\n",
    "\n",
    "根据文章进行分组， 对于每篇文章的用户， 计算点击的时间间隔\n",
    "将用户的数量取倒数， 然后用户的数量和时间间隔归一化， 然后相加得到热度特征， 该值越小， 说明被点击的次数越大且时间间隔越短， 文章比较热\n",
    "当然， 这只是给出一种判断文章热度的一种方法， 这里大家也可以头脑风暴一下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "uuid": "b1113c07-c221-489a-9539-a3599193d865"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>click_article_id</th>\n",
       "      <th>user_num</th>\n",
       "      <th>time_diff_mean</th>\n",
       "      <th>hot_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>69</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>94</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>125</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   click_article_id user_num  time_diff_mean hot_level\n",
       "0                 3        1             0.0         1\n",
       "1                69        1             0.0         1\n",
       "2                84        1             0.0         1\n",
       "3                94        1             0.0         1\n",
       "4               125        1             0.0         1"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def hot_level(all_data, cols):\n",
    "    \"\"\"\n",
    "    制作衡量文章热度的特征\n",
    "    :param all_data: 数据集\n",
    "    :param cols: 用到的特征列\n",
    "    \"\"\"\n",
    "    data = all_data[cols]\n",
    "    data.sort_values(['click_article_id', 'click_timestamp'], inplace=True)\n",
    "    article_hot = pd.DataFrame(data.groupby('click_article_id', as_index=False)[['user_id', 'click_timestamp']].\\\n",
    "                               agg({'user_id':np.size, 'click_timestamp': {list}}).values, columns=['click_article_id', 'user_num', 'click_timestamp'])\n",
    "    \n",
    "    # 计算被点击时间间隔的均值\n",
    "    def time_diff_mean(l):\n",
    "        if len(l) == 1:\n",
    "            return 1\n",
    "        else:\n",
    "            return np.mean([j-i for i, j in list(zip(l[:-1], l[1:]))])\n",
    "        \n",
    "    article_hot['time_diff_mean'] = article_hot['click_timestamp'].apply(lambda x: time_diff_mean(x))\n",
    "    \n",
    "    # 点击次数取倒数\n",
    "    article_hot['user_num'] = 1 / article_hot['user_num']\n",
    "    \n",
    "    # 两者归一化\n",
    "    article_hot['user_num'] = (article_hot['user_num'] - article_hot['user_num'].min()) / (article_hot['user_num'].max() - article_hot['user_num'].min())\n",
    "    article_hot['time_diff_mean'] = (article_hot['time_diff_mean'] - article_hot['time_diff_mean'].min()) / (article_hot['time_diff_mean'].max() - article_hot['time_diff_mean'].min())     \n",
    "    article_hot['hot_level'] = article_hot['user_num'] + article_hot['time_diff_mean']\n",
    "    \n",
    "    article_hot['click_article_id'] = article_hot['click_article_id'].astype('int')\n",
    "    \n",
    "    del article_hot['click_timestamp']\n",
    "    \n",
    "    return article_hot\n",
    "article_hot_fea = hot_level(all_data, ['user_id', 'click_article_id', 'click_timestamp'])    \n",
    "article_hot_fea.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "37235f89-2744-428d-b0b0-50b9242ce121"
   },
   "source": [
    "#### 用户的系列习惯\n",
    "这个基于原来的日志表做一个类似于article的那种DataFrame， 存放用户特有的信息, 主要包括点击习惯， 爱好特征之类的\n",
    "\n",
    "用户的设备习惯， 这里取最常用的设备（众数）\n",
    "用户的时间习惯： 根据其点击过得历史文章的时间来做一个统计（这个感觉最好是把时间戳里的时间特征的h特征提出来，看看用户习惯一天的啥时候点击文章）， 但这里先用转换的时间吧， 求个均值\n",
    "用户的爱好特征， 对于用户点击的历史文章主题进行用户的爱好判别， 更偏向于哪几个主题， 这个最好是multi-hot进行编码， 先试试行不\n",
    "用户文章的字数差特征， 用户的爱好文章的字数习惯\n",
    "这些就是对用户进行分组， 然后统计即可"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "72acd64a-3a30-43c8-a0ed-dca3727136c5"
   },
   "outputs": [],
   "source": [
    "def device_fea(all_data, cols):\n",
    "    \"\"\"\n",
    "    制作用户的设备特征\n",
    "    :param all_data: 数据集\n",
    "    :param cols: 用到的特征列\n",
    "    \"\"\"\n",
    "    user_device_info = all_data[cols]\n",
    "    \n",
    "    # 用众数来表示每个用户的设备信息\n",
    "    user_device_info = user_device_info.groupby('user_id').agg(lambda x: x.value_counts().index[0]).reset_index()\n",
    "    \n",
    "    return user_device_info\n",
    "# 设备特征(这里时间会比较长)\n",
    "device_cols = ['user_id', 'click_environment', 'click_deviceGroup', 'click_os', 'click_country', 'click_region', 'click_referrer_type']\n",
    "user_device_info = device_fea(all_data, device_cols)\n",
    "user_device_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "317abb59-85f2-4700-a3b0-6611fee27044"
   },
   "outputs": [],
   "source": [
    "def user_time_hob_fea(all_data, cols):\n",
    "    \"\"\"\n",
    "    制作用户的时间习惯特征\n",
    "    :param all_data: 数据集\n",
    "    :param cols: 用到的特征列\n",
    "    \"\"\"\n",
    "    user_time_hob_info = all_data[cols]\n",
    "    \n",
    "    # 先把时间戳进行归一化\n",
    "    mm = MinMaxScaler()\n",
    "    user_time_hob_info['click_timestamp'] = mm.fit_transform(user_time_hob_info[['click_timestamp']])\n",
    "    user_time_hob_info['created_at_ts'] = mm.fit_transform(user_time_hob_info[['created_at_ts']])\n",
    "\n",
    "    user_time_hob_info = user_time_hob_info.groupby('user_id').agg('mean').reset_index()\n",
    "    \n",
    "    user_time_hob_info.rename(columns={'click_timestamp': 'user_time_hob1', 'created_at_ts': 'user_time_hob2'}, inplace=True)\n",
    "    return user_time_hob_info\n",
    "user_time_hob_cols = ['user_id', 'click_timestamp', 'created_at_ts']\n",
    "user_time_hob_info = user_time_hob_fea(all_data, user_time_hob_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "1108ca88-e9ab-4beb-aeda-8418f452bd59"
   },
   "source": [
    "#### 用户的主题爱好\n",
    "这里先把用户点击的文章属于的主题转成一个列表， 后面再总的汇总的时候单独制作一个特征， 就是文章的主题如果属于这里面， 就是1， 否则就是0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "7994607f-d601-465e-8a98-42b98b8c833c"
   },
   "outputs": [],
   "source": [
    "def user_cat_hob_fea(all_data, cols):\n",
    "    \"\"\"\n",
    "    用户的主题爱好\n",
    "    :param all_data: 数据集\n",
    "    :param cols: 用到的特征列\n",
    "    \"\"\"\n",
    "    user_category_hob_info = all_data[cols]\n",
    "    user_category_hob_info = user_category_hob_info.groupby('user_id').agg({list}).reset_index()\n",
    "    \n",
    "    user_cat_hob_info = pd.DataFrame()\n",
    "    user_cat_hob_info['user_id'] = user_category_hob_info['user_id']\n",
    "    user_cat_hob_info['cate_list'] = user_category_hob_info['category_id']\n",
    "    \n",
    "    return user_cat_hob_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "c80f5fec-c11f-457e-b1fc-10081b94cb20"
   },
   "outputs": [],
   "source": [
    "user_category_hob_cols = ['user_id', 'category_id']\n",
    "user_cat_hob_info = user_cat_hob_fea(all_data, user_category_hob_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "51e024ba-a728-47fa-8ab4-054237728af7"
   },
   "source": [
    "### 用户的字数偏好特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "a35e6724-1ec0-425b-91a3-016162c5beb0"
   },
   "outputs": [],
   "source": [
    "user_wcou_info = all_data.groupby('user_id')['words_count'].agg('mean').reset_index()\n",
    "user_wcou_info.rename(columns={'words_count': 'words_hbo'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "cdf2da81-baef-4895-b5b8-dd0762e9ac0a"
   },
   "source": [
    "#### 用户的信息特征合并保存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "79236863-574f-4af5-b53b-7f39d87f18d1"
   },
   "outputs": [],
   "source": [
    "# 所有表进行合并\n",
    "user_info = pd.merge(user_act_fea, user_device_info, on='user_id')\n",
    "user_info = user_info.merge(user_time_hob_info, on='user_id')\n",
    "user_info = user_info.merge(user_cat_hob_info, on='user_id')\n",
    "user_info = user_info.merge(user_wcou_info, on='user_id')\n",
    "# 这样用户特征以后就可以直接读取了\n",
    "user_info.to_csv(save_path + 'user_info.csv', index=False)   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "0b98678f-1151-46a2-a9c5-4930e0160a23"
   },
   "source": [
    "#### 用户特征直接读入\n",
    "如果前面关于用户的特征工程已经给做完了，后面可以直接读取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "19485d7d-3911-405a-9985-2ba6ded4d051"
   },
   "outputs": [],
   "source": [
    "# 把用户信息直接读入进来\n",
    "user_info = pd.read_csv(save_path + 'user_info.csv')\n",
    "if os.path.exists(save_path + 'trn_user_item_feats_df.csv'):\n",
    "    trn_user_item_feats_df = pd.read_csv(save_path + 'trn_user_item_feats_df.csv')\n",
    "    \n",
    "if os.path.exists(save_path + 'tst_user_item_feats_df.csv'):\n",
    "    tst_user_item_feats_df = pd.read_csv(save_path + 'tst_user_item_feats_df.csv')\n",
    "\n",
    "if os.path.exists(save_path + 'val_user_item_feats_df.csv'):\n",
    "    val_user_item_feats_df = pd.read_csv(save_path + 'val_user_item_feats_df.csv')\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "# 拼上用户特征\n",
    "# 下面是线下验证的\n",
    "trn_user_item_feats_df = trn_user_item_feats_df.merge(user_info, on='user_id', how='left')\n",
    "\n",
    "if val_user_item_feats_df is not None:\n",
    "    val_user_item_feats_df = val_user_item_feats_df.merge(user_info, on='user_id', how='left')\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "    \n",
    "tst_user_item_feats_df = tst_user_item_feats_df.merge(user_info, on='user_id',how='left')\n",
    "trn_user_item_feats_df.columns\n",
    "Index(['user_id', 'click_article_id', 'sim0', 'time_diff0', 'word_diff0',\n",
    "       'sim_max', 'sim_min', 'sim_sum', 'sim_mean', 'score', 'rank', 'label',\n",
    "       'click_size', 'time_diff_mean', 'active_level', 'click_environment',\n",
    "       'click_deviceGroup', 'click_os', 'click_country', 'click_region',\n",
    "       'click_referrer_type', 'user_time_hob1', 'user_time_hob2', 'cate_list',\n",
    "       'words_hbo'],\n",
    "      dtype='object')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "688f9e97-44e1-4e6b-b5b7-4585da5d6ea8"
   },
   "outputs": [],
   "source": [
    "# 拼上文章特征\n",
    "trn_user_item_feats_df = trn_user_item_feats_df.merge(articles, left_on='click_article_id', right_on='article_id')\n",
    "\n",
    "if val_user_item_feats_df is not None:\n",
    "    val_user_item_feats_df = val_user_item_feats_df.merge(articles, left_on='click_article_id', right_on='article_id')\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "\n",
    "tst_user_item_feats_df = tst_user_item_feats_df.merge(articles, left_on='click_article_id', right_on='article_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "9b9e585b-5709-46be-b214-d3e8dae93410"
   },
   "source": [
    "#### 召回文章的主题是否在用户的爱好里面"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "bdc8fee2-5133-43be-8257-5cd4b28f6c38"
   },
   "outputs": [],
   "source": [
    "trn_user_item_feats_df['is_cat_hab'] = trn_user_item_feats_df.apply(lambda x: 1 if x.category_id in set(x.cate_list) else 0, axis=1)\n",
    "if val_user_item_feats_df is not None:\n",
    "    val_user_item_feats_df['is_cat_hab'] = val_user_item_feats_df.apply(lambda x: 1 if x.category_id in set(x.cate_list) else 0, axis=1)\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "tst_user_item_feats_df['is_cat_hab'] = tst_user_item_feats_df.apply(lambda x: 1 if x.category_id in set(x.cate_list) else 0, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "705a24b8-035c-431a-aa2a-207aa781cbce"
   },
   "outputs": [],
   "source": [
    "# 线下验证\n",
    "del trn_user_item_feats_df['cate_list']\n",
    "\n",
    "if val_user_item_feats_df is not None:\n",
    "    del val_user_item_feats_df['cate_list']\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "    \n",
    "del tst_user_item_feats_df['cate_list']\n",
    "\n",
    "del trn_user_item_feats_df['article_id']\n",
    "\n",
    "if val_user_item_feats_df is not None:\n",
    "    del val_user_item_feats_df['article_id']\n",
    "else:\n",
    "    val_user_item_feats_df = None\n",
    "    \n",
    "del tst_user_item_feats_df['article_id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "c2ea7599-9c7d-4e22-be6f-dc97e78b2810"
   },
   "source": [
    "#### 保存特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "54967ff0-4c84-4ef6-8634-3c320a52d9dc"
   },
   "outputs": [],
   "source": [
    "# 训练验证特征\n",
    "trn_user_item_feats_df.to_csv(save_path + 'trn_user_item_feats_df.csv', index=False)\n",
    "if val_user_item_feats_df is not None:\n",
    "    val_user_item_feats_df.to_csv(save_path + 'val_user_item_feats_df.csv', index=False)\n",
    "tst_user_item_feats_df.to_csv(save_path + 'tst_user_item_feats_df.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "markdown",
    "uuid": "b5c14f92-a949-4f7c-8c05-c88d6b8b43ed"
   },
   "source": [
    "### 总结\n",
    "\n",
    "特征工程和数据清洗转换是比赛中至关重要的一块， 因为数据和特征决定了机器学习的上限，而算法和模型只是逼近这个上限而已，所以特征工程的好坏往往决定着最后的结果，特征工程可以一步增强数据的表达能力，通过构造新特征，我们可以挖掘出数据的更多信息，使得数据的表达能力进一步放大。 在本节内容中，我们主要是先通过制作特征和标签把预测问题转成了监督学习问题，然后围绕着用户画像和文章画像进行一系列特征的制作， 此外，为了保证正负样本的数据均衡，我们还学习了负采样就技术等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "d0cf5a07-6b30-499a-9b22-e9ce11afd49e"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
